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IMAGE COMPENSATION APPARATUS 



BACKGROUND OF THE INVENTION 
The present invention relates to an image compensation 
5 apparatus, and particularly to an image compensation apparatus 
for compensating an attitude of image pickup section capable 
of rotating around three axes (roll axis, pitch axis and yaw 
axis) . ' “ ' • 

In recent years, a technique for assisting pilot control. 
10 by acquiring image information about the outside world in flight 
by a predetermined image pickup device and generating outside: 
world video based on this image information and displaying the 
generated outside world video oh a predetermined display is 
proposed and commercialized. 

15 For example, a technique in which a video camera (for 

example, see patent reference 1 ) mounted in a gyroscopic driving 
device capable of rotating around mutually orthogonal, three 
axes and arbitrarily changing an image pickup direction is 
attached with an outside of aircraft is proposed. When such 
20 a gyroscopic driving device is adopted, even in case that the 
aircraft encounters gust, outside world video is maintained 
in a stable state by gyro stabilizing effect. 

On the other hand, in the case of controlling an aircraft, 
a technique for providing outside world video (image 
25 information) which changes in response to movement of a head 
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or a sight line of a pilot while properly showing an inclination 
of the horizon or a flight direction as well as the stable outside 
world video described above is required. 

As the technique for providing such image information, 

5 a system (hereinafter called "an enhanced vision system") in 
which a small display (Head Mounted Display: hereinafter called 
"HMD") is mounted on a head of a pilot and also a camera is 
installed on an aircraft through a predetermined driving device 
and. a sight, line direction of the pilot is detected by measuring . 

10 movement of the. head of the pilot .and the camera is ; driven 
according' to the detected sight, line direction'is proposed (for. 
example, see. patent reference 2). 

[Patent Reference 1] 

JP-A-5-167901 (Page 2, Fig. 2) 

.15 • [Patent' Reference 2 ] • . ) ..... •• 

JP-A-2001-344597 (Page 1, Fig. 1) 

However, the driving device used in the image information 
providing system as described above has a structure in which 
a camera can be rotated only around two axes . (pitch axis and . 
20 yaw axis) , so that a pilot (or a co-pilot) visually identifying 
image information providedby this system may have physiological 
discomfort . 

For example, when an aircraft moves to pitching direction 
in a state in which a pilot turns toward the side temporarily 
25 and thereby a camera rotates around a yaw axis, the image 
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information is displayed as if the aircraft had performed roll 
movement. On the other hand, since the aircraft actually does 
not perform the roll movement, the pilot feels that the aircraft 
performs the pitching movement by vestibular organs made of 
5 semicircular canals and ear stones. As a result of this, a 
contradiction between image information obtained from eyes by 
the pilot and space information obtained from the vestibular 
organs arises, so that the pilot tends to get travel-sick. 

Such a phenomenon . becomes remarkable in the case that 
10 . the pilot: cannot see the outside at all or in the case that 
it is difficult for the pilot to see the outside (for example/ 
at the time of night or bad weather conditions in the case of 
using a see-through visual display system).. 

15- SUMMARY OF THE INVENTION . 

An object of the invention is to solve physiological 
discomfort of a pilot caused by a contradiction between image 
information obtained from eyes by the pilot and space information 
obtained from another organ in an image compensation apparatus 
20 for controlling an attitude of image pickup section used in 
an image information providing system of an aircraft. 

In order to solve the object, the invention as claimed 
in aspect 1 is characterized in that in an image compensation 
apparatus for controlling an attitude of image pickup section 
25 which is mounted in an aircraft and can rotate around a roll 
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axis, a pitch axis and a yaw axis, the apparatus comprises sight 
line direction information detection section for detecting 
information about a sight line direction of a pilot, driving 
section which outputs a driving signal for rotating and driving 
5 the image pickup section with reference to the information about 
the sight line direction and rotates and drives the image pickup 
section based on this driving signal, image pickup attitude 
information detection section for detecting information about 
an attitude of the image pickup section,- aircraft attitude. 
10 -information detection section for detecting. information about 
an attitude of . the- aircraft, and correction section for 
correcting the driving signal with reference to the information 
about the attitude of the image pickup' section and the 
information about the attitude of the aircraft . - •• • 

15 According, to the invention as claimed in aspect 1, by 

the correction section, the driving signal for rotating and 
driving the imagepickup section can be corrected with reference 
• to the information about the attitude of the image pickup section 
detected by the image pickup attitude information detection 
20 section and the information about the attitude of the aircraft 
detected by the aircraft attitude information detection 
section. 

For example, when an aircraft performs pitching movement 
in a state in which a pilot turns toward the side temporarily 
25 and thereby the image pickup section rotates around a yaw axis 
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and the image pickup section shifts sideward, a driving signal 
for rotating and driving the image pickup section around a roll 
axis can be corrected based on the information about the attitude 
of the image pickup section (yaw angle information about the 
image pickup section) and the information about the attitude 
of the aircraft (pitch angle information about the aircraft) . 
Therefore, even when the aircraft performs the pitching movement , 
image information in which the roll axis hardly moves can be 
displayed. . As a result. of this, a contradiction betweenvimage 
information obtained from eyes by the! pilot and space information 
obtained f rom vestibular organs can be solved, so that the pilot 
can. be prevented' from getting travel-sick. . . .• >• 

The invention as claimed in aspect 2 is. characterized 
in that in the image compensation apparatus as claimed in aspect 
1,- the . image pickup section is a stereo camera, constructed of 
two cameras placed at a predetermined spacing. 

The invention as claimed in aspect 3 is characterized 
in that in the image compensation apparatus as claimed in aspect 
1 or 2, the information about the attitude of the image pickup 
section is yaw angle information about the image pickup section 
and the information about the attitude of the aircraft is pitch 
angle information about an aircraft and the correction section 
corrects a driving signal for rotating and driving the image 
pickup section around a roll axis. 

The invention as claimed in aspect 4 is characterized 




in that in the image compensation apparatus as claimed in any 
one of aspects. 1 to 3, there is provided driving prevention 
section for preventing rotation and driving of the image pickup 
section when the information about the sight line direction 
5 does not reach a predetermined level. 

According to the invention as claimed in aspect 4, there 
is provided driving prevention section for preventing rotation 
and driving of the image pickup section when the information 
about .the sight line direction does not reach a predetermined 
10 level,, so that when a pilot does not desire a Ghange. in image 
information,' a situation in which the image pickup section 
rotates and the image information changes. due to small movement 
of an aircraft or the pilot can be prevented. Therefore,., stable 
: image information can be provided for the pilot, so that the 
15 pilot . does not have ..physiological : discomfort . 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig- 1 is a block diagram describing a configuration of 
an image compensation apparatus according to an embodiment of 
20 the invention. 

Fig. 2 is a schematic diagramof a stereo camera controlled 
by the image compensation apparatus shown in Fig. 1. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
25 An embodiment of the invention will be described below 
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in detail based on the drawings. An image compensation 
apparatus used in an image information providing system of an 
aircraft will be described in the embodiment. Incidentally, 
the image information providing system comprises an HMD mounted 
on a head of a pilot and a stereo camera mounted in an aircraft, 
and the image compensation apparatus according to the embodiment 
functions so as to; control an attitude of the stereo camera. 

First, a configuration of the image compensation 
apparatus according, to the embodiment will be described using 
Figs. 1 and/2. -Fig. .1 .is a block diagram. . describing the 
configuration; of." the image compensation apparatus, and Fig. 

2 is a schematic diagram of image pickup section (stereo camera 
100) controlled. by the image compensation apparatus . shown, in 
: Firg; 1 

The image compensation apparatus is an . apparatus for 
controlling an attitude of the stereo camera 100 (see Fig. 2) 
capable of rotating around a roll, axis A r , a pitch axis A 2 and 
a yaw axis A 3 orthogonal mutually. The stereo camera 100 is . 
constructed of two cameras 110 placed at a predetermined spacing 
und is installed in the lower front of a fuselage B of an aircraft 
through a stereo camera driving device 200. 

As shown in Fig. 1, the image compensation apparatus 
comprises sight line direction information detection section 
10, a computation device 20, the stereo camera driving device 
200, image pickup attitude information detection section 30, 




aircraft attitude information detection section 4 0, correction 
section 50, driving prevention section 60 and so on. 

The sight line direction information detection section 
10 is means for detecting information about a sight line 
5 direction of a pilot. In the embodiment, information (head 
roll angle information <E> T , head yaw angle information *Pt and 
head pitch angle information ® T ) about an attitude of a head 
of the pilot is adopted as the "information about the sight 
line... direction" . .. The information about the: attitude of the 
10 head of the: pilot detected by ' the sight line . direction 
information detection section 10 is outputted to the computation 
device 20. • •••: =•••• •. •••.•... ■. ■ 

The sight line 1 direction information detection section. 
10 can be constructed of an HMD 70 mounted, on the head of the 
15 pilot and a three-axis gyro sensor mounted in the. fuselage;. 
Further, the sight line direction information detection section 
■10 can be constructed. of a sensor (a magnetic sensor, a light, 
wave sensor, a super-sound wave sensor, etc. ) mounted in the 
HMD 70 and signal (a magnetic signal, a light wave, signal, a 
20 super-sound wave signal, etc.) generation section mounted in 
the aircraft . 

The computation device 20 is a device for outputting a 
driving signal for rotating and driving the stereo camera 100 
with reference to the information about the sight line direction 
25 detected by the sight line direction information detection 
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section 10. The computation device 20 comprises a low-pass 
filter21, limiters 22, A/Dconverters 23, the correction section 
50, the driving prevention section 60 and so on. 

The low-pass, filter 21 is a filter for eliminating 
5 high-frequency noise by passing only a low-frequency portion 
(about several tens Hz or lower) of the information (head roll 
angle information <J> T / head yaw angle information T t and head 
pitch angle information 0 T ) detected by the sight line direction 
information detection section 10 , and eliminates :an inf luence 
10 of •.vibration of the fuselage. 

. • : The limiter 22 performs a function of limiting a rotational 

angle of the camera 110 in order to prevent the two cameras 
. 110 of the stereo .camera 100 from abutting on the fuselage B-; 

By the limiters. 22, a rotational angle (roll angle <J>) around. 
15 the roll axis . Ai, a rotational angle (pitch angle 0) around; 
the pitch axis A 2 and a rotational angle (yaW angle 'F) around 
the yaw axis A 3 of the stereo camera 100 are limited to a range., 
in which the stereo camera 100 does not abut on the fuselage 
B. Incidentally, ranges of these roll angle <I>, pitch angle 
20 .0 and yaw angle are determined by a mounting position of 

the stereo camera driving device 200 with respect to the fuselage 
B. 

Incidentally, the case that the camera 110 of the right 
side with respect to a flight directionof the fuselage B inclines 
25 downward from a predetermined reference position is set to a 
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roll angle of "plus"/ and the case that the two cameras 110 
turn upward from a predetermined reference position is set to 
a pitch angle of "plus"/ and the case that the two cameras 110 
turn rightward from a predetermined reference position is set 
5 to a yaw angle of "plus." . 

. The ‘ stereo camera driving device 200 is a device for 
changing an attitude of the stereo camera 100 by rotating, and 
driving the stereo camera 100 based on the driving signal 
outputted. by the computation device 20 Incidentally, the 
10 computation device 2 0 and the stereo -camera driving device 200 
construct, driving section in the invention. . r . • - ' ' 

The image pickup attitude information detection. section 
30 . is means for detecting information about an attitude of the 
stereo camera 100. in the embodiment, yaw: angle information 
15 (a rotational angle around the- yaw axis A3) about the stereo 

camera 100 is adopted as the "information about an attitude 
of the stereo camera 100". 

The aircraft attitude information detection section 40 . 
is means for detecting information about an attitude of the 
20 aircraft. In the embodiment, pitch angle information © B (an 
angle between a horizontal plane and a back-and-forth axis of 
the fuselage) about the fuselage is adopted as the "information 
about an attitude of the aircraft". An inertial device using 
gyro or a device using electromagnetic waves such as light or 
25 radio waves can be adopted as the image pickup attitude 
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information detection section 30 and the aircraft attitude 
information detection section 40. 

The correction section 50 is means for correcting the 
driving signal for rotating and driving the stereo camera 100 
5 based on the information about an attitude of the stereo camera 
100. (yaw angle information 'F about the stereo camera 100) and 
the information about an attitude of the aircraft, (pitch angle 
information 0 B about the fuselage) and is provided inside the 
computation device 20 . . •• v •; .• • • ... . 

10 • v : The correction section 50 in the embodiment' calculates, 

correction roll angle information <E> F by calculating correction 
angle information- :$ c .using the yaw angle information W- about 
■ the stereo. camera 100 and the pitch angle information 0 B about 
the -fuselage and subtracting this correction angle, information 
15 <D C from the' head roll angle information O t detected by the sight 
line direction information detection section 10. That is, the 
following formula is obtained. 

Op - Ot~^C ( 1 ) 

Here, the correction angle information 3> e is calculated by the 
20 following formula. 

Oc = ©Eisin 1 ? (2) 

Incidentally, the pitch angle information 0 B about the fuselage 
and the yaw angle information about the stereo camera 100 
are converted into digital signals by the A/D converters 23. 
25 The driving prevention section 60 is means for preventing 
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rotation and driving of the stereo camera 100 when the 
information about a sight line direction detected by the sight 
line direction information detection section 10 does not reach 
a predetermined level, and is provided inside the computation 
5 device 20. In the case of making a description specifically, 
the driving prevention section 60 prevents an output of. a driving 
signal to the stereo camera driving device 200 when absolute 
Values of the head roll angle information <J> T , the head yaw angle 
information T t and the head pitch angle informat ion-0 T :detected 
10 by the -.sight line: direction information detection: section- 10 
.are less than ■ definition values <J> 0 , To, ©o predefined: : 

' Next, an attitude control action of the. stereo camera. 
100 by the image compensation apparatus according to the 
embodiment -will be described. • 

15 *-'M '••• First, when a -head of a pilot turns sideward, • the. sight 

line direction information detection section 10 detects 
information (head yaw angle information T t ) about an attitude 
of the head of the pilot. The head yaw angle information T t 
detected by the sight line direction information detection 
20 section 10 is outputted to the computation device 20 and reaches 
the driving prevention section 60 through the low-pass filter 
21. When an absolute value of the head yaw angle information 
T t is more than or equal to a definition value T 0 in the driving 
prevention section 60, the head yaw angle information Tt is 
25 transmitted to the stereo camera driving device 200 through 
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the limiter 22 as it is. On the other hand, when the absolute 
value of the head yaw angle information T t is less than the 
definition value T 0 , the driving prevention section 60 prevents 
an output of a driving signal to the stereo camera driving device 
5 200 . 

. When the head yaw. angle information T t passing the driving 
prevention section 60 is inputted to the stereo camera driving 
device 200 , the stereo camera driving device 200 rotates the 
stereo camera 100 around the yaw axis A 3 based on this head. 
10 yaw angle information T t . When the stereo camera 100 rotates 
around the. yaw axis A 3 , the image pickup attitude. information 
detection section 30 detects yaw angle information T about the 
stereo camera 100.. Then,, the yaw angle information T detected 
by the image pickup attitude information detection section 30 
15 is converted into a digital signal by the A/D converter .23. arid 
thereafter is transmitted to the correction section 50 of the 
computation device 20 . 

When an aircraft performs pitching movement in a state 
in which the stereo camera 100 rotates around the yaw axis A 3 
20 thus, the aircraft attitude information detection. section 40 
detects pitch angle information © B about a fuselage. The pitch 
angle information 0 B detected by the aircraft attitude 
information detection section 40 is transmitted to the 
correction section 50 of the computation device 20 through the 
25 A/D converter 23. 
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The correction section 50 calculates correction roll 



angle information <I> F with reference to the yaw angle information 
'P detected by the image pickup attitude information detection 
section 30 and the pitch angle information ©b detected by the 
5 aircraft attitude information detection section 40, and 
corrects a driving signal for rotating and driving the stereo 
camera 100. 

For example, when head roll angle information O t detected 
by the sight line direction information detection section- 10 
10 is 0° (no inclination to the right and left) and the yaw angle 
information 'P detected by the image pickup attitude information 
detection section 30 is 30° (rightward) and the pitch angle 
information 0 B detected by the aircraft attitude information 
detection section 40 is 30° (upward) , the. correction roll angle 
15 information O f is calculated as the following formula using 
the formula (1) and formula (2) described above. 

<D f = <I> T -0 B sin v P = 0-30xsin30° = -15 (° ). (3) 

That is, the head roll angle information O t detected is 
0° and if this information <$ T is inputted to the stereo camera 
20 driving device 200 as it is, a head roll <1> of the stereo camera 
100 does not incline from a predetermined reference position 
but in the embodiment, the correction roll angle information 
<E>f (“15° ) calculated by the correction section 50 is inputted 
to the stereo camera driving device 200, so that the stereo 
25 camera 100 inclines 15° to the left side from the predetermined 
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reference position. As a result of this, even when the aircraft 
performs the pitching movement, image information in which a 
roll axis hardly moves can be displayed. 

In the image compensation apparatus according to the 
5 embodiment described above, by the correction section 50, a 
driving signal can be corrected with reference to the yaw angle 
information 'P about the stereo camera 100 detected by the image 
pickup attitude information detection section 30 and the pitch 
angle information ©b about the fuselage detected by .the aircraft 
10 attitude information'- detection section 40. - 

"• That is, when the aircraft performs the pitching movement 
in a state in which- a pilot turns toward the side temporarily ■ 
and thereby the stereo camera 100 rotates around the yaw axis 
A 3 and shifts sideward, the amount of rotation and. driving around 
15 the roll axis Ai of the stereo camera 100 can be : corrected-by 
calculating the correction roll angle information ® F based on 
the yaw. angle information about the stereo camera 100 and 
the pitch angle information © B about the fuselage. Therefore, - 
the pilot does not have an illusion that the aircraft performs 
20. roll movement. As a result of this, a contradiction between 
image information obtained from the HMD 70 by the pilot and 
space information obtained from vest ibular organs can be solved, 
so that the pilot can be prevented from getting travel-sick. 

Further, in the image compensation apparatus according 
25 to the embodiment, rotation and driving of the stereo camera 
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100 can be prevented by the driving prevention section 60 when 
absolute values of information (the head roll angle information 
Ot, the head yaw angle information 'Ft and the head pitch angle 
information 0 T ) detectedby the sight line direction information 
detection section 10 are less than definition values (<E> 0 , 'Fo , 
®o J- predefined. Therefore, when the pilot does not desire a 
change in image information, a situation in which the stereo 
camera 100 rotates and the image information changes due to 
variations in a. sight line direction caused by vibration of 
the fuselage or resonance of a sensor can be prevented. As 
a result of this, stable image informat ion can be provided, for 
the pilot, so that the pilot does not- have physiological 
discomfort including recognition of a feeling of fatigue. . 

Incidentally, in the embodiment described, above, means 
for detectinginformation about an attitude of a head of a pilot 
is adopted as the sight line direction information detection 
section 10, but an eye camera for detecting movement of eyeballs 
of the pilot may also be adopted as the sight line direction 
information detection section 10. 

Further, the aircraft in which the image compensation 
apparatus according to the invention is mounted is a manned 
aircraft at which a pilot boards, but a kind of its aircraft 
is not limited. That is, the image compensation apparatus 
according to the invention can be mounted in various manned 
aircraft such as a fixed-wing aircraft, a rotary-wing aircraft 




or an airship. 

Further, the image compensation apparatus according to 
the invention can be applied to an aircraft for performing remote 
control above ground. For example, a ground operator wears 
5 a display such as the HMD 70 mounting the sight, line direction 
information detection section 10 and devices other than this 
display are mounted in. a fuselage and thereby operation can 
be performed. 

.According to the invention as claimed in aspect 1, by 
10 correction section, a .driving signal can be corrected with 
reference to 'information about an attitude of image pickup 
section-detected by image pickup attitude information-detection 
section and information about an attitude of an aircraf t detected 
by aircraft : attitude . information detection .section... For 
15 example, when .an aircraft performs pitching movement in a state 
in which a pilot turns toward the side temporarily and thereby 
the image. pickup section shifts sideward; the amount of rotation 
and driving around a roll axis of the image pickup section can 
be corrected based on a yaw angle of the image pickup section 
20 and a pitch angle of the fuselage. As a result of this, image 
information can be displayed without producing an illusion that 
the aircraft performs roll movement, so that a contradiction 
between image information obtained by the pilot and space 
information obtained from vestibular organs can be solved and 
25 the pilot can be prevented from getting travel-sick. 
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According to the invention as claimed in aspect 4, there 
is provided driving prevention section for preventing rotation 
and driving of image pickup section when information about a 
sight line direction does not reach a predetermined level, so 
5 that when a pilot does not desire a change in visibility 
information; a situation in which the image' pickup section 
rotates and the. image information changes due to variations 
in the sight line direction caused by vibration of a fuselage 
or resonance of .a sensor can be prevented. Therefore; . stable • 
10 image inf ormation can be provided for the pilot, j so •• that the • 
pilot does : not have physiological discomfort including 
recognition -of a feeling of fatigue. 
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